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A REPORT TO 

THE KANSAS COUNCIL OF INSTRUCTIONAL ADMINISTRATORS 

ON 

INSTITUTIONAL EFFECTIVENESS ASSESSMENT 

AND 

STUDENT ACADEMIC ACHIEVEMENT 



The North Central Association of Colleges and Schools serves as the 
regional accrediting agency for the community colleges in Kansas. 
The purpose of North Central is to examine institutional effective- 
ness in order to certify that institutions are deserving of 
accreditation. In 1989 the North Central Association initiated 
efforts to encourage colleges to incorporate into their 
institutional effectiveness assessment plans a component dealing 
with student academic achievement. In October of 1989, the 
Ccnmission on Institutions of Higher Education approved the 
following statement on behalf of the North Central Association: 

The Commission wants to make clear that all institutions 
are expected to assess the achievement of their students. 
With this statemient -ve . make explicit -the : ^Commission's ^ 
position that student achievement i6 a critical cbmpbneht 
in assessing overall institutional effectiveness. Our 
expectation is that an institution has and is able to 
describe a program by which it documents student academic 
achievement. 

In addition to the efforts of North Central to promote 
institutional effectiveness assessment, community colleges are 
responding to the directives of the Committee of Practitioners 
mandated by the Carl Perkins Legislation. The System of Measures 
and Standards of Performance that have been articulated by this 
committee will require community college career programs to report 
on the successful completion of vocational-technical competencies 
and workplace competencies by career program completers. These 
reported data will have to be validated by some methods of 
assessment of student achievement. 

In responding to the guidelines of both North Central and the 
reporting requirements of Carl Perkins legislation, community 
colleges will be offered the opportunity to engage in significant 
and meaningful review of the curriculum and the processes by which 
they deliver instruction. The assessments made, the data 
collected, the plans that result can help college faculty 
understand more clearly the impact of their instructional efforts 
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and guide attempts to continuously improve and enhance curricula. 
If the results of these processes are used in the preparation of 
budgets and the reallocation of resources, colleges will be able to 
get maximum impact out of the resources available for instruction. 
The data resulting from assessment, if considered in context and 
appropriately used, can increase credibility with external 
legislative and funding agencies and give the consumer increased 
confidence in the effectiveness of educational efforts. The effect 
of institutional effectiveness assessment will be to document 
accountability, increase professionalism, enhance curricula and 
instruction, assist in recruiting both students and faculty,, and 
guide colleges in more efficient budgeting and planning. In a time 
of limited resources this process can restore credibility to 
education and significantly assist community colleges in fulfilling 
their missions and goals. 

In response to current efforts to establish institutional 
effectiveness plans and implement outcomes-based student 
achievement measurements, the Kansas Council of Instructional 
Administrators appointed a subcommittee to study the status of | 
institutional effectiveness assessment in Kansas community ■ 
colleges. This subcommittee consisted of Bob Paxton, Cindy Hoss, | 
Daun Anderson, and Ken Gibson. The subcommittee met in December of 
1991. At that meeting the subcommittee outlined two tasks: first, 
the subcommittee would research the literature on institutional 
effectiveness assessment to provide a bibliography oa the subject 
for community college deans , of instruction; second, the- sub-- 
•committee would • send - a • letter-- to Kansats- Goribnuriity cdlle^e' dean^ 
requesting a descriptidn of practices being utilized thirbughdut the 
state. Bob Paxton and Cindy Hoss agreed to do the search of the 
literature and Ken Gibson agreed to solicit input from the deans of 
instruction and to use that input to construct a questionnaire to 
be sent to all colleges during the summer of 1992. After some I 
consideration. Bob Paxton and Cindy Hoss recommended that rather I 
than preparing an annotated bibliography, the Kansas Council of I 
Instructional Administrators should purchase copies of the League 
for Innovation abstract entitled: Assessing Institutional 
Effectiveness in Community Colleges , 1990. Those abstracts have 
been purchased and are included as an appendix to this report. 

All of the deans of instruction in Kansas community colleges 
received a request for information on each college's institutional 
effectiveness assessment plans. Responses were received from 10 of 
the 19 colleges. Based on an analysis of those responses, Dr. 
Gibson created a questionnaire to solicit more specific information 
from the colleges. During July and August, 19 questionnaires were 
sent to the deans of instruction in Kansas community colleges. 
Each of the 19 colleges responded for a response ratp. of 100%. 
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THE FINDINGS 



ASSESSMENT OF INSTITUTif||Jp^^^^g^''S13^ 

In order to identify assessment techniques and collect data, it is 
necessary to first define the criteria against which performance is 
to be measured. The League for Innovation abstract points out the 
importance of a clearly defined and well understood mission 
statement as the initial step in the assessment process. The 
responses to question #1 suggest that Kansas community colleges 
recognize the importance of the mission statement to the assessment 
process. Seventy -nine per cent of the colleges indicated that they 
had engaged in a review of ihe mission statement as a preparation 
for assessing institutional effectiveness. Responses to the second 
part of the question indicate that most of the important 
constituencies of the college participated in the mission review. 
However, it should be noted that less than half of the colleges 
indicated student involvement in the review process. 



1. Has your college conducted a formal review of its mission 
statement as part of the institutional effectiveness 
assessment process? 





•RESPONDfiD . 


PERCENTAGE 


YES 


15 


79% 


NO 


4 


21% 


If yeS/ indicate which of the following groups were involved 
in the review process. 




RESPONDED 


PERCENTAGE 


Board of Trustees 


13 


87% 


College Administration 


15 


100% 


Faculty 


15 


100% 


Student Services 


14 


93% 


Students 


7 


47%, 


Community Members 


9 


60% 



The literature on institutional effectiveness assessment makes it 
very clear that this process must be supported by the various 
constituencies at the college if it is to be successful. The 
responses to question #2 suggest that the process may face some 
difficulty in Kansas community colleges without an increase in 
faculty support. There is an obvious gap between the perception of 
the commitment of administration/board and the perception of 
commitment by the faculty . In the majority of colleges , the 
administrators and board members are viewed as "strongly committed" 
to this process of institutional effectiveness assessment, but only 
a third or less of the colleges view the faculty and community 
members as "strongly committed." This finding would suggest that 
there needs to be much more discussion with faculty, students, and 
members of the r omraunity on the importance of assessment in light 
of accreditation standards, current legislation, and emphases on 
accountability and quality. Efforts to improve instruction through 
assessment will only be successful if the faculty and the community 
perceive their value and support their impact. 



2. Which of the following individuals or groups would you 
characterize as **strongly committed" to the assessment of 
institutional effectiveness* 





RESPONDED 


RANK 


Board of Trustees 


11 


4 


President 


13 


■ 3 


Chief Instructional Officer 


18 


1 


Chief Student Affairs Officer 


15 


2 


Faculty 


6 


5 


Community Members 


2 


6 


Other: Director of Planning 


1 


7 



The importance of institutional effectiveness to community colleges 
is reflected by the responses to question #3, Seventy-four per 
cent of the colleges indicated that they had appointed a special 
task force and/or committee to deal with the issue. However, it is 
important to note the lack of participation by students and 
community members in this process. This might suggest that the 
movement to involve business people and other community members in 
the improvement of education may need to be promoted more 
extensively. 




3 • Has your college appointed an institutional task force or 
committee to guide the college's efforts in institutional 
assessment? 





RESPONDED 


PERCENTAGE 


YES 


14 


74% 


NO 


5 


26% 


If yes, how were the members chosen? 




RESPONDED 


PERCENTAGE 


Appointed 


9 


64% 


Nominated 


1 


7% 


Volunteer 


1 


7% 


Election 


1 


7% 


Which of the following groups are represented by one or more 
members? 




. RESPONDED 


PERCENTAGE: . 


Administration 


12 


86% 


Faculty 


13 


93% 


Student Services 


13 


93% 


Institutional Research 


6 


43% 


Students 


4 


29% 


Community 


1 


7% 


Other: Classified Staff /Trustees 


2 


14% 



The responses to question #4 suggest that a number of colleges have 
yet to meet the requirement of North Central that a strategic 
institutional effectiveness plan be created. Less than 50% of the 
colleges indicated that they had a formal plan or model. 
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4- Which of the following best describes your college's current 
strategic plan for assessing your institution's effectiveness? 





RESPONDED 


PERCENTAGE 


A written formal plan or model has 
been developed 


8 


42% 


An informal plan has been developed; 
nothing in writing 


2 


11% 


Nothing has been developed yet; we're 
working on it 


6 


32% 


No plans to develop an assessment 
plan at the present time 


2 


11% 



Much of the current discussion of effectiveness assessment deals 
with efforts at the institutional level; however, for college-wide 
assessment to be effective, the various programs must be actively 
involved in the process* The responses to questions #5 and #6 
suggest that slightly over half of the colleges currently do 
effectiveness assessment at the program level, but those that do 
not are planning to implement such processes ♦ 



5« Does your college have committees charged with assessing 
institutional effectiveness at the program level? 





RESPONDED 


PERCENTAGE 


Yes, for some programs 


3 


16% 


Yes, for all programs 


7 


37% 


No 


8 


42% 


Not Applicable 


1 


5% 1 


6* If no, when do you anticipate appointing a task force for this 
purpose? 




RESPONDED 


PERCENTAGE 


Fall 1992 


4 


44% 


Spring 1993 


1 


11% 


No plans to initiate such a task force 
at this time 


3 


33% 


Other 


1 


11% 
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While assessment and evaluation are not necessarily the same 
activity and while current program evaluations are not necessarily 
outcomes-based, the responses to question #7 suggest that academic 
programs are evaluated regularly in Kansas community colleges • 
These evaluation processes might serve as a starting point for the 
implementation of outcomes-based assessment* The responses to 
question #7 suggest that academic program evaluation is extensive 
in Kansas community colleges; however, the responses to question #8 
suggest that these evaluations may not be student outcomes based, 
for until faculty can reach agreement on intended student outcomes, 
it will be difficult to assess student academic achievement. 



7« Does your college conduct formal evaluations of acadeaic 
programs? 





RESPONDED 


PERCENTAGE 


Yes, annually 


5 


26% 


Yes, bi -annually 


1 


5% 


Yes, every five years 


6 


32% 


Yes, for accreditation/no regular 
schedule /as needed 


3 


16% 


No 


5 


26% 


8. Have faculty in these prograiiis reached written agreement oh 
the intended student outcomes for the programs? 




RESPONDED 


PERCENTAGE 


Yes 


3 


16% 


Draft in progress 


6 


32% 


No 


9 


47% 


Not Applicable 


1 ; 5% 



The responses to question #9 suggest some attempt to use 
standardized tests to meet the requirement of student achievement 
assessment; however, the results may confirm a level of discomfort 
on the part of colleges and especially faculty to depend on 
standardized tests for outcomes measurement* There has been much 
discussion in the literature concerning the advantages and 
disadvantages of using standardized tests, but colleges will need 
to proceed with caution in this area if they want to promote 
faculty support for an outcomes-based curriculum. 



9« Do you use any of the following standardized measures of 
student academic achievement as part of your program 
evaluation? 





RESPONDED 


PERCENTAGE 


College Assessment of Academic 
Proficiency 


4 


21% 


College Base 


1 


5% 


Other: Asset (ACT) 


1 


5% 



The responses to question #10 indicate that the most often used 
measures of student achievement in Kansas community colleges are 
retention rates, grade distribution analysis, license examinations, 
employment rates, and transfer data from four-year colleges and 
universities, A second level of instruments consists of competency 
based profiles, attitudinal and/or opinion surveys, program 
specific student withdrawal analysis, graduate profiles, and 
instruments developed by and used in specific departments for all 
sections of a course. Finally, those assessment techniques used 
least of all are the locally constructed tests developed for 
college-wide use, capsbone courses and/or experiences, and 
portfolios. The most often used techniques appear to result from 
•practices that vW^ire already in place prior to the emphasis on 
effectiveness assessment. Most colleges have .been/' oing retention 
studies^ gr.ade distribution analysis, and collect ina information 
from transfer institutions for a number of years • In those 
programs in which national or state registry examinations are 
required, these results have been important to community colleges 
for a number of years also. The problem with employment and 
transfer data, however, as always has been the lack of numbers to 
make possible significant statistical analysis. In addition. North 
Central has insisted that if colleges use grades as an indicator of 
student academic achievement, the faculty must deliberate and 
determine standardized criteria and rationale for grades so that a 
grade in one course or section implies the same level of competency 
as the same grade in another course or section. While grades have 
always been an indicator of student success, there have been 
difficulties in comparison across individual course lines. If 
students are getting better grades, can we isolate what increased 
learning contributes to that phenomenon? Data from the colleges to 
which our students transfer would appear to be a valuable measure 
of student success; however, it has been difficult to get such data 
in a complete and meaningful format so that we can make conclusions 
with confidence. 
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10. Which of the following methodologieo do you use to arsesa 
academic progr2uu effectiveness? 





RESPONDED 


PERCEf/TAGE 


Attitudinal and/or opinion surveys 


9 




Locally constructed test(s) developed 
for college-wide use 


3 


16% 


Competency based student profiles 


10 


53% 


Capstone courses and/or experiences 


3 


16% 


Portfolios 


3 


16% 


License examinations 


13 


68% 


EinployiP.ent rates 


12 


63% 


Transfer data from 4-year colleges or 
universities 


11 


58% 


Retention rate 


14 


74% 


Grade distribution analysis 


14 


74% 


Program-specific student withdrawal 
analysis 


7 


37% 


Graduate profiles 


■6 ■ ■ 


32% 


Instruments developed by and used in 
specific departments for all sections 
of a course 


6 


32% 



The responses to questions #11, #12, /'13 deal with those non- 
academic areas of the college such as student services, 
administrative services, institutional research, etc. '/There such 
evaluation is conducted, the process involves collecting opinions 
reported on surveys or exit interviews. The responses suggest that 
there has been little attempt to c;et staff in those areas to 
articulate intended outcomes statements. 
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11* Does your college conduct formal evaluations of non«*academic 
programs? 





RESPONDED 


PERCENTAGE 


Yes, annually 


4 


21% 


Yes, bi-annually 


1 


5% 


Yes, every five years 


3 


16% 


Yes, Student Support Services/every 10 
years for NCA 


2 


11% 


No 


8 


42% 


Not Applicable 


1 


5% 


Have staff members in these areas reached written agreement on 
the intended outcomes for their progrzuss? 




RESPONDED 


PERCENTAGE 


Yes 


1 


5% 


Draft in progress 


3 


16% 


No 


■ ■, ' ■■ i3" ■ 


.68%. 



13. If yeS/ which of the following research methodologies are used 
to measure these non-^academic progrzua outcomes? 





RESPONDED 


PERCENTAGE 


Attitudinal surveys 


6 


32% 


Exit interviews 


4 


21% 


Quantitative data on tasks performed 


3 


16% 



A caveat might be offered to Kansas community colleges as a result 
of the responses to question #14. Approximately 67% of the 
colleges indicated that they considered the evaluation of faculty 
to be part of the institutional effectiveness assessment process. 
On page 48 of Assessing Institutional Effectiveness in Community 
Colleges, the authors state: "Assessment results should certainly 
not be used as part of faculty evaluation..." Given the results of 
question #2 which suggest that faculty are not highly committed to 
this process; given the results of question #8 which indicate that 
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much work has yet to be done to get faculty to articulate intended 
student outcomes for academic programs; given that the results of 
question #16 suggest that getting and maintaining faculty support 
is one of the major barriers to the success of outcomes assessment, 
it would appear to be important to make clear from the very 
beginning that outcomes assessment is not another name for faculty 
evaluation. The community colleges in Kansas have a faculty 
evaluation process which is mandated by state statute. They do not 
need for effectiveness assessment *io be a part of that process. If 
proficiency statements are kept at the course (where multi- 
sectioned), program, degree, or college level, individual faculty 
will not be threatened. It is important to realize that if the 
assessment results are not satisfactory there are many factors 
which impact those measurements^ including facilities, resources, 
technology, faculty, entry level skills of students, just to 
mention a few. 



14. Do you consider the evaluation of individual faculty and/or 
other personnel to be part of your institutional effectiveness 
process? 





RESPONDED 


PERCENTAGE 


Yes 


12 


63% 


No. 


6 


32% 



The responses to question #15 indicate that the Kansas community 
colleges take seriously the charge to create institutional 
effectiveness plans and to measure student achievement and that 
they want to learn more about how to accomplish these objectives. 



15. Should there be a stata-vide resource 'or sharing assessment 
plans and/or experiences? 





RESPONDED 


PERCENTAGE 


Yes 


19 


100% 


No 


0 


0% 




If yes, please indicate which of the following you feel would 
be most effective at accomplishing this objective. 





RESPONDED 


PERCENTAGE 


Create a position within the state 
ciepairT.iuenT. or eaucauion 


1 


5% 


Create a permanent resource center at 
the state department level 


4 


21% 


Plan a state-wide annual conference on 
institutional effectiveness 


16 


84% 


Publish a newsletter to facilitate 
sharing ideas, experiences, and 
research results 


11 


58% 


Other: Use KACC 


1 


5% 



The responses to question #16 clearly demonstrate the importance of 
faculty participation in any institutional effectiveness plan or 
student outcomes measurement. The majority of community colleges 
clearly recognizes that faculty support is essential to the success 
of any plan. Articulating program and student oxitcomes and 
designing appropriate methods of data collecting will be 
significant barriers to success without faculty support. The five 
barriers that were most identified as impeding success revolved 
around the faculty t r activities that must be performed by faculty. 
In comparison to the problem of obtaining faculty participation, 
obtaining appropriate funding, student participation, and 
administrative support were seen as less important problems. 



In creating exit level competencies as criteria against which our 
efforts can be measured, faculty will be taking advantage of the 
opportunity to define themselves, their efforts, and their 
institutions. It is better that faculty of community colleges 
define what their curricula are designed to do and validate their 
success in accomplishing those goals than that they wait for 
external agencies to mandate regional and national standardized 
testing which may negatively impact their efforts. A further 
caveat in this area concerns the underprepared students. Those who 
promote outcomes-based curricula should realize that if colleges 
are to enable students to reach agreed upon exit level standards, 
they may have to be more rigid in the entry level standards they 
set for programs. 
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16. In general/ all of the following are important when conducting 
an institutional effectiveness assessment. Given the goal of 
obtaining useable information for improving both the product 
and the process of education, which of the following represent 
significant obstacles or barriers to successful implementation 
of an institutional effectiveness assessment plan on your 
campus « 





RESPONDED 


PERCENTAGE 


Articulating measurable intended 
program outcomes 


11 


58% 


Articulating measurable student 
achievement outcomes 


12 


63% 


Obtaining initial faculty support 


11 


58% 


Sustaining long-term faculty 
commxumenc 


14 


74% 


Obtaining continuing administrative 
support 


6 


32% 


Designing appropriate methods of data 
collection/analysis 


11 


58% 


Motivating students to contribute 
valid efforts/ information 


7 


37% 


Obtaining appropriate funding 


9 


47% 


Other: 1) Obtaining faculty/admin- 
istrative consensus when faced with 
negatives; 2) Lack of top-level 
leadership 


3 


16% 



This study suggests that Kansas community colleges are committed to 
institutional effectiveness assessment, are searching for ways to 
initiate outcomes-based measurement, are willing to document 
success and validate performance when such activities lead to 
credibility and professional accountability. It also appears to 
suggest that the colleges would be receptive to guidance, 
direction, and the sharing of efforts in this area. All 19 of the 
colleges agreed that there should be some state-wide resource for 
sharing assessment plans and/or experiences; the ^^ast majority 
thought this should take the form of an annual conference, and all 
colleges indicated that they would definitely send representatives 
to such a conference. This response might suggest that there is a 
significant opportunity for the Competency-based Center at Washburn 
University to be more proactive in assisting community colleges 
v;ith this issue. 



17. Would your college send representatives to a state-vide 
conference on institutional effectiveness and student outcomes 
assessment? 





RESPONDED 


PERCENTAGE 


Yes 


19 


100% 


No 


0 


0% 
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